Crowd and AI Powered Manipulation: Characterization and Detection

User reviews are ubiquitous. They power online review aggregators that influence our daily-based decisions, from what products to purchase (e.g., Amazon), movies to view (e.g., Netflix, HBO, Hulu), restaurants to patronize (e.g., Yelp), and hotels to book (e.g., TripAdvisor, Airbnb). In addition, policy makers rely on online commenting platforms like Regulations.gov and FCC.gov as a means for citizens to voice their opinions about public policy issues. However, showcasing the opinions of fellow users has a dark side as these reviews and comments are vulnerable to manipulation. And as advances in AI continue, fake reviews generated by AI agents rather than users pose even more scalable and dangerous manipulation attacks. These attacks on online discourse can sway ratings of products, manipulate opinions and perceived support of key issues, and degrade our trust in online platforms. Previous efforts have mainly focused on highly visible anomaly behaviors captured by statistical modeling or clustering algorithms. While detection of such anomalous behaviors helps to improve the reliability of online interactions, it misses subtle and difficult-to-detect behaviors. This research investigates two major research thrusts centered around manipulation strategies. In the first thrust, we study crowd-based manipulation strategies wherein crowds of paid workers organize to spread fake reviews. In the second thrust, we explore AI-based manipulation strategies, where crowd workers are replaced by scalable, and potentially undetectable generative models of fake reviews. In particular, one of the key aspects of this work is to address the research gap in previous efforts for anomaly detection where ground truth data is missing (and hence, evaluation can be challenging). In addition, this work studies the capabilities and impact of model-based attacks as the next generation of online threats. We propose inter-related methods for collecting evidence of these attacks, and create new countermeasures for defending against them. The performance of proposed methods are compared against other state-of-the-art approaches in the literature. We find that although crowd campaigns do not show obvious anomaly behavior, they can be detected given a careful formulation of their behaviors. And, although model-generated fake reviews may appear on the surface to be legitimate, we find that they do not completely mimic the underlying distribution of human-written reviews, so we can leverage this signal to detect them

Search Rank Fraud De-Anonymization in Online Systems

We introduce the fraud de-anonymization problem, that goes beyond fraud detection, to unmask the human masterminds responsible for posting search rank fraud in online systems. We collect and study search rank fraud data from Upwork, and survey the capabilities and behaviors of 58 search rank fraudsters recruited from 6 crowdsourcing sites. We propose Dolos, a fraud de-anonymization system that leverages traits and behaviors extracted from these studies, to attribute detected fraud to crowdsourcing site fraudsters, thus to real identities and bank accounts. We introduce MCDense, a min-cut dense component detection algorithm to uncover groups of user accounts controlled by different fraudsters, and leverage stylometry and deep learning to attribute them to crowdsourcing site profiles. Dolos correctly identified the owners of 95% of fraudster-controlled communities, and uncovered fraudsters who promoted as many as 97.5% of fraud apps we collected from Google Play. When evaluated on 13,087 apps (820,760 reviews), which we monitored over more than 6 months, Dolos identified 1,056 apps with suspicious reviewer groups. We report orthogonal evidence of their fraud, including fraud duplicates and fraud re-posts.Comment: The 29Th ACM Conference on Hypertext and Social Media, July 201

TOmCAT: Target-Oriented Crowd Review Attacks and Countermeasures

Online platforms like Amazon, Yelp, and Regulations.gov give a voice to masses of users through reviews, comments, and ratings. However, this crowd-based feedback is susceptible to manipulation. To tackle this problem, most previous efforts have only indirectly sought to uncover targets of attacks by focusing on manipulation at the review or user level. Instead, this paper focuses on the challenge of countering target-oriented crowd attacks. We introduce a unique ground truth dataset of Amazon products that have been targeted for attack and identify two target-oriented attack patterns: (i) promotion attacks and (ii) restoration attacks. With these attacks in mind, we propose the TOmCAT detection framework based only on the timing and sequencing of product ratings. Although TOmCAT succeeds in uncovering targets of manipulation with high accuracy by addressing existing attacks, strategic attackers potentially can create hard-todetect behavioral patterns by undermining timing-based footprints. Hence, we further propose a complementary approach to TOmCAT called TOmCATSeq which is resistant against strategic manipulation

Behavioral Analysis of Review Fraud: Linking Malicious Crowdsourcing to Amazon and Beyond

We exploit the prevalence of malicious review writers on crowdsourcing platforms like RapidWorkers to identify actual fraud reviews on Amazon. Complementary to previous efforts which often rely on proxies for fraud reviews, we present a long-term study of actual fraudulent behaviors in online review manipulation. We find that these malicious reviewers — though often providing seemingly legitimate opinions — do exhibit significant differences from normal reviewers in terms of ratings distribution, length of the reviews, and the burstiness of the reviews themselves. We additionally study the evolution of these reviews, and find striking temporal changes that could support future discovery of these reviewers who may be “hiding in plain sight.

A Large-Scale Study of ISIS Social Media Strategy: Community Size, Collective Influence, and Behavioral Impact

The Islamic State of Iraq and Syria (ISIS) has received a tremendous amount of media coverage in the past few years for their successful use of social media to spread their message and to recruit new members. In this work, we leverage access to the full Twitter Firehose to perform a largescale observational study of one year of ISIS social activity. We quantify the size of ISIS presence on Twitter, the potential amount of support it received, and its collective influence over time. We find that ISIS was able to gain a relatively limited portion from the total influence mass on Twitter and that this influence diminished over time. In addition, ISIS showed a tendency towards attracting interactions from other similar pro-ISIS accounts, while inviting only a limited anti-ISIS sentiment. We find that 75% of the interactions ISIS received on Twitter in 2015 actually came from eventually suspended accounts and that only about 8% of the interactions they received were anti-ISIS. In addition, we have created a unique dataset of 17 million ISIS-related tweets posted in 2015 which we make available for research purposes upon request